The Beast of Aggregating Cognitive Load Measures in Technology-Based Learning

An increasing part of cognitive load research in technology-based learning includes a component of repeated measurements, that is: participants are measured two or more times on the same performance, mental effort or other variable of interest. In many cases, researchers aggregate scores obtained from repeated measurements to one single sum or average score per participant and use these aggregated scores in subsequent analysis. This paper demonstrates some dangers of this commonly encountered aggregation approach and presents two comprehensive alternatives: Split-plot analysis of variance (ANOVA) and more flexible two-level regression analysis. The core message of this paper is that the application of the aggregation approach can seriously distort our view of effects and relations of interest and should therefore not be used in cognitive load research. Multilevel analysis of repeated measurements data can account for various features of the data and constitutes a best practice.